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The problem of measuring SES on assessments 



Much attention has been given to the effects of SES on achievement and educational 
attainment, particularly during the last half of the 20* century. Recognition that achievement 
tends to vary by socioeconomic status became policy in the form of the No Child Left Behind 
Act. The No Child Left Behind Act (NCLB) of 2002 ushered in a new era for U.S. schools. 
Schools are faced with increasingly stringent achievement goals, not only for their student 
populations as a whole, but across subgroups. One of these subgroups is the “economically 
disadvantaged” (Rhode Island Department of Elementary and Secondary Education, 2004) which 
is often measured by eligibility for free and reduced lunch. Within states, NCLB requires steady 
improvements in achievement within these subgroups, thus requiring states to report 
achievement by socioeconomic status. 

Though there has been much debate about the merits of NCLB and the standardized tests 
that have become increasingly emphasized (Kohn 2001 provides a call to arms against 
standardized tests), the links between socioeconomic status and achievement have been well 
established. Socioeconomic status will remain an important background variable for researchers 
and thus they should be provided with the best possible measures. Unfortunately, the current 
measures that are used often have many problems. Above and beyond the statistical concerns 
regarding validity in measurement, on a practical level, if one is going to ensure that 
economically disadvantaged groups are progressing, researchers need to ensure that they are 
identifying all members of the groups. 

The current measures of SES on some large scale assessments and that are used by 
researchers in other contexts are problematic. On a national level, NCLB requires NAEP results 
to be reported by socioeconomic status of students, however, NAEP does not have a parental 
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questionnaire which would enable researchers to gather this information directly from students’ 
parents and guardians. Similarly, international assessments such as TIMSS or PISA do not use 
parent questionnaires. Thus, reports of socioeconomic status either have to come from the 
students themselves or from a proxy measure such as free and reduced lunch. Much research 
has been done investigating the links between family socioeconomic status and educational 
attainment, and the links between school effects and achievement (Buchmann, 2002). There has 
also been much research done on measuring the socio-economic status of adults (Buchmann, 
2002; Duncan & Peterson, 2001; Hauser & Warren, 1997; Smith, 1994). What has not been 
addressed as rigorously is how to assess the socioeconomic status of the child when their parents 
can not be queried. A key part of the problem is the need to balance the very real concerns of 
cost and obtrusiveness with the need to collect valid data. 

In this article, we review some of the ways socioeconomic status has been measured on 
assessments and the issues associated with measuring SES of students, issues which are not 
limited to statistical concerns. We also present possible proxy measures that could be used as a 
means of potentially overcoming some of the problems with current measures of SES. Any 
proposed measures would need to be tested for validity and reliability so we also present 
elements for a study assessing the construct validity of aggregate measures of SES as proxies for 
student SES on assessments. We use NAEP and TIMSS as examples, however, these approaches 
could be applied to other assessments and research contexts. 

Current Measures of SES on Assessments 

Among researchers there is some consensus that measures of socioeconomic status 
should be comprised of three theoretical concepts: educational attainment, occupational status, 
and financial resources (Baer, Baldi, & Merola 2005). Currently, a common measure of 
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socioeconomic status used by educational researchers is eligibility for free and reduced price 
lunch. This has been used by researchers in a variety of contexts, not just on assessments (Kurki, 
Boyle, & Aladjem, 2005) and could be considered a proxy measure either for poverty or 
financial resources. This measure suffers from a variety of problems. The use of the federal free 
and reduced price lunch program as a proxy for poverty has been attacked because enrollment 
rates are low, they decline with grade level, and because the eligibility requirements are based on 
crude measures of income and family size (Hauser & Carr, 1995). In addition, data on relative 
pecuniary indicators such as the poverty level or participation in the free or reduced priced lunch 
program are problematic because definitions and criteria for inclusion may change over time, 
making comparisons difficult. Some scholars also maintain that the threshold levels for relative 
indicators fail to sufficiently capture people in the lower end of the SES spectrum (Hauser, 

1994). Additionally, students classified as eligible may actually not be, due to a rule that if a 
certain percentage of students in a school are eligible the school is allowed to make all students 
eligible. 

Aside from the issues of how to determine eligibility, even if all students who are indeed 
eligible are enrolled and standards didn’t change over time, the measure would still have 
problems. Free and reduced price lunch only measures one dimension of socio-economic status 
and does not capture the full range of financial resources available to all students in a population. 
The free and reduced lunch measure fails to provide information about students whose fa mi lies 
are in the middle and upper ends of the income distribution. Similarly eligibility for free and 
reduced price lunch, since it is a threshold measure and students whose families are at 185% of 
the poverty level are eligible, individual students that qualify for this program could be at very 
different income levels from each other (Kurki et. al, 2005). So arguably, even if it were not 
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degrading as a measure due to the problems with reporting and declining enrollment rates, it 
would not be a fully valid measure of SES. 

Eligibility for free and reduced price lunch does have the benefit of being an unobtrusive 
measure of SES, allowing researchers to avoid the thorny issue of asking students about the 
socioeconomic status of their fa mi lies. Some researchers are able to ask parents about their 
family’s socioeconomic status and in this case they can draw on the extensive research that has 
been done on quering adults about their socioeconomic status. Researchers have developed some 
guidelines for questions that should be administered to families (primarily to parents) to assess 
family socioeconomic status in research on child development (Entwisle & Astone, 1994; Hauser 
1994). Hauser (1994) and Entwisle and Astone (1994) disagree about the usefulness and ease of 
gathering occupational information. Hauser (1994) and Entwisle and Astone (1994) do agree, 
however, that asking about financial resources is problematic. Duncan and Peterson (2001) 
address the question of gathering financial information from families and disagree with Hauser 
(1994) and Entwisle and Astone (1994) that it is problematic, however, they don’t address the 
issue of gathering financial information from students when the parents can not be queried. 
Studies of student achievement also vary in the extent to which they address these facets of SES, 
with some only measuring one or two facets (Buchmann, 2002). 

Some large scale assessments use measures that have questionable validity and/ or don’t 
address all the facets of SES. Currently, NAEP collects data on two of the three SES dimensions: 
educational attainment and financial resources. These constructs are measured on NAEP 
through student reports of the educational attainment of their parents and information from 
schools about whether sampled students participate in the federal free and reduced price lunch 
program. TIMSS asks about parents education and assets as a proxy for financial resources. 



5 




Though this approach is consistent with that used in other assessments (Buchmann, 2002) 
concerns have been raised about the data collected from measures that query the students about 
their family’s socio-economic status. 

One of the arguments is that student reports of the educational attainment of their parents 
may be inaccurate, especially among younger children. There is also concern over the 
intrusiveness of asking students about the educational attainment of their parents. In the context 
of NAEP, posing questions to students about the educational attainment of their parents and 
gathering data from schools about free and reduced price lunch participation may be inconsistent 
with NAGB guidelines about avoiding intrusive questions on NAEP. This problem is not simply 
an American phenomenon. International assessments such as TIMSS have faced the problem of 
some countries not allowing questions about parental occupation and wealth.. Eimiting the 
measurement of socio-economic status to financial resources and educational attainment also 
means that the occupational status facet of the socioeconomic status construct is not being 
addressed. 

In sum, as researchers we are faced with measures of socio-economic status on some 
large scale assessments do not fully measure the construct of SES and data collected from the 
measures that may be of questionable validity. Indeed, in the case of free and reduced price 
lunch, the measure may be degrading. Concurrently, there has been a call for reporting 
achievement by SES, though the current measures may not be up to the task. If achievement is 
going to accurately and (detailed) reported by SES, better measures of SES must be developed. 

Measuring Socio-economic Status 

The individual components of SES, (educational attainment, occupational status, and 
financial resources) have been extensively researched and discussed over time, in conjunction 
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with how these facets can be combined to make a composite measure of SES. (Baer, Baldi, & 
Merola 2005). However, researchers seem to differ on what components they use in their 
analyses (Buchmann 2002 provides some good examples). Indeed, some researchers prefer to 
look at the facets of SES individually to determine how much each component contributes to any 
observed effects. Since any composite is only as good as the components that make it up, 
making sure that the individual components are measured correctly is (key). Over time and 
extensive research, some broad guidelines that have been developed for measuring the 
educational attainment, occupational status, and financial resources of adults. . 

Educational Attainment 

Educational attainment is generally measured in one of two ways within the sociological 
literature: (a) asking parents how many years of education they have completed or (b) asking 
them about the highest degree they have attained (allowing options for people who pursued but 
did not complete a degree) (Smith, 1994). Given that eligibility for many jobs is contingent on 
the completion of a degree and that personal prestige often changes upon completion of a degree, 
the second option may be a better measure. Research into the advantages of using either the 
number of years of education or the highest degree attained as a measure of educational 
attainment indicate that highest degree tends to have a higher association with SES, in particular 
with income and occupational prestige (Smith, 1994). 

An issue with some international surveys is to make levels of schooling comparable 
across countries. This has led to the development of the International Standard Classification of 
Education (UNESCO 1997) and the Comparative Analysis of Social Mobility in Industrial 
Nations (CASMIN ) (Mueller & Karle, 1993) These scales allow for comparability across 
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countries. Though these may be better measures of education for the assessing of SES, they still 
don’t address whether students can actually answer the questions or not. 

Occupational status 

Virtually all composite measures of SES draw on occupational data as a key construct in 
defining the indicator (Nam, 2000). To develop a complete picture of a respondent’s working life 
one must know the respondent’s occupation, occupational status (i.e., self-employed vs. working 
for someone else), and employment sector (i.e., government, private industry, ete.) (e.g., Kreiger 
et. al, 1997; Hauser & Warren, 1997). Typically, open-ended questions are used to gather this 
information and there is agreement about the need to ask multiple questions to get the best data 
(Hauser, 1994; Entwisle & Astone, 1994). These questions, which ask respondents about their 
occupations and the industries in which they work, tend to have low rates of refusal and non- 
response because respondents perceive little risk in providing this information (Hauser & 

Warren, 1997). This approach is used by such surveys as the General Social Survey, United 
States Census, the Panel Study of Income Dynamics, the 1992 National Adult Eiteracy Survey, 
and the 1998 National Educational Eongitudinal Study. 

Transferring this approach from adults to students may present a problem. There have 
been concerns about confidentiality (Buchmann, 2002) and whether students can accurately 
answer these questions (Entwisle & Astone, 1994). As indicated earlier, TIMSS does not have an 
occupation question because some countries prohibit it (Buchmann, 2002). As a compromise, 
some categorical questions have been created that older students can answer (Strieker. . .) This 
multiple choice format also didn’t conform to recommendations about gathering occupational 
data. 
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The Program for International Student Assessment (PISA), which is an international 
assessment given to 15-year olds, in 2000 did use open ended response questions which were 
subsequently tested in four of countries to see if students could answer the questions. Validity 
studies were conducted in Canada, the Czech Republic, France and the United Kingdom to test the 
reliability of student answers to the questions on their parents’ occupations. These studies found a 
correlation of between .70 and .86 between the students’ responses and the parents’ responses to the 
open-ended occupational questions. This was considered high since the test/retest correlation of 
parents who were asked about their occupations on two occasions was in this range. (Adams & Wu, 
2002) 

Hauser (1994) also indicates that students ages 14 and 15 can answer these occupational 
questions about as well as adults. It’s unclear though whether younger students can answer these 
questions and more research needs to be done in this area. 

Financial Resources 

The financial resources available to families and households may be characterized in 
terms of earned income, assets, or total wealth (income plus assets, minus debt), though income 
often receives more attention by social scientists. There is some debate, however, about 
respondents’ abilities to answer questions about income and wealth. Some argue that unlike their 
reactions to questions dealing with occupational status, respondents may be far more likely to 
refuse to answer questions about their financial resources (Hauser & Warren, 1997; Entwisle & 
Astone, 1994). Furthermore, surveys that query respondents about income are time-consuming 
and difficult to conduct, both because many participants are reluctant to report their income and 
because income may fluctuate rapidly over time (Hauser, 1994; Krieger et. ah, 1997). In 
addition, the data tend to suffer from poor recall and unreliability (Hauser & Warren, 1997). 
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Duncan and Peterson (2001) argue, however, that by both creating questions that are 
conceptually appropriate and instituting proper interviewing techniques and ways of easing 
respondents burdens, income data can be collected without nonresponse or validity problems. 

Though there is some debate about the ability to obtain valid data, measuring financial 
resources is important because such resources are a direct means of acquiring goods and 
services; moreover, we concur with Duncan and Peterson (2001) that financial resources are best 
characterized by the total wealth a family possesses. As noted above, wealth is considered to be 
the net value of income plus assets, minus the family’s debt (Duncan & Peterson, 2001). 
Although income and assets are positively correlated, they are distinct types of financial 
resources. For example, Duncan and Peterson provide the example of elderly persons whose 
income is low but whose home equity is high due to their having paid off a mortgage and 
benefiting from rising home values. The most common form of assets in the U.S. is home equity. 

Given that adults potentially have problems answering these questions, conventional 
wisdom is that students will not be able to answer these questions (see Hauser 1994; Entwisle & 
Astone, 1994). Response rates may be low and the validity of the responses may be suspect 
(Buchmann, 2002). Also, in cases such as TIMSS, asking about this information directly has 
been prohibited (Buchmann, 2002). This has led researchers to develop proxies such as using 
free and reduced lunch which was mentioned earlier. Another approach has been to measure 
family wealth through indices of home possessions (Buchmann, 2002). The list of possessions 
will be includes assets that are not necessarily educational resources, such as recreational 
vehicles and video cameras, for example. Using assets a indicator for wealth has a long 
theoretical history going back to Veblen’s idea of Conspicuous Consumption (Veblen, 1899). 
These lists may be improved by adding a question about home ownership. 
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Aggregate Measures of SES 

Given the problems that students may have in answering SES questions, and that parents 
often can not be queried either due to cost or concerns about obtrusiveness, another way of 
measuring SES needs to be considered. Ideally, SES measures for students participating in 
NAEP or other assessments could be collected unobtrusively from a secondary data source 
without troubling students or their parents. Unfortunately, such a data source is not available at 
the individual level. Though individual level data is unavailable, aggregated data from a 
secondary source can be used as a proxy for the household SES of sampled students in 
assessments. Entwisle and Astone (1994) suggest using this method for finding home prices and 
Kurki and her colleagues (2005) have attempted to use this to measure poverty levels, but this 
approach can be used to gather other SES information as well. 

Aggregate measures of SES have both conceptual and methodological advantages over 
individual or household indicators in certain studies. Conceptually, they enable the researcher to 
account for environmental factors that operate beyond the individual and that influence the 
distribution of resources. Eor example, the rate of poverty in a neighborhood may affect the life 
chances of individuals in a community independently of household poverty rates, as the 
aggregate indicator can serve as a proxy for the general level of safety and well-being in the 
neighborhood (Krieger et ah, 1997). Methodologically, aggregate measures are often preferred 
because of the difficulty of collecting SES data, especially concerning income, from respondents. 

Scholars can select from a variety of aggregate levels of varying sizes. With Census data, 
the tract level contains an average of 4,000 residents, the block group an average of 1,000, and 
the block an average of about 85 residents. Though smaller levels of aggregation may be 
preferred, little SES data is collected at the block unit in order to protect the confidentiality of 
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respondents (Krieger et al., 1997). Depending on the level of aggregation, the researcher can 
select from a range of aggregate SES indicators pertaining to educational attainment, 
occupational status, and financial resources. 

One of the problems with using aggregate proxies is that comparisons of individual SES 
indicators with their aggregate-level proxies have revealed that the latter are not as precise as the 
former (Geronimus & Bound, 1998; Soobader et al., 2001). Interestingly, smaller levels of 
aggregation do not necessarily reduce their bias, nor is the bias systematically oriented upward or 
downward. Eor example, in a study evaluating health outcomes, Soobader et al. (2001) 
concluded that tract-level assessments of income underestimated income to a greater extent than 
those based on postal codes. Yet for education, the zip codes evidenced a greater bias than both 
the tract and block group. Clearly, more research is necessary in order to determine the degree to 
which aggregate-level measures either understate or overstate individual measures of SES, as 
well as how these biases can best be managed. 

As indicated, one of the benefits of the Census measures in comparison to the current 
NAEP SES measures is that these measures cover more facets of the SES domain, and indeed, 
cover all of the three key constructs the literature indicates as important. Eigure 1 presents a 
comparison of the measures currently in NAEP and the proposed aggregate measures. As 
demonstrated in Eigure 1, there are noticeable gaps in the measurement of the key SES 
constructs on NAEP. NAEP measures do not represent all of the SES constructs identified in the 
literature as important. Noticeably lacking is any measure of occupational status. 

In addition, the scope of the current measures is limited in regard to their coverage of the 
SES constructs of educational attainment, and financial resources. Besides it’s other problems, 
use of the measure “eligibility for free and reduced price lunch” allows researchers to make 
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inferences about students whose family income is below the threshold, however, given that the 
range of incomes above the point of eligibility is extensive , little can be inferred about this 
population. For example, if one wanted to determine whether the increase in achievement by 
income is a linear one, this can not be fully assessed with the current measure. The range of 
questions that can be answered increases dramatically with the addition of family income, and 
assets as measures of financial resources 

Similarly, Census data provide information about a greater range of educational outcomes 
than the current NAEP measures of parents’ educational attainment. The current measures on 
NAEP provide information about whether parents have less than a high school education, 
completed high school, have some education after high school, or if they graduated from college. 
The Census asks about educational attainment in finer detail, asking about what grades were 
completed if a person did not receive a high school diploma, and then asks about degrees other 
than a college degree, such as graduate degrees and associates degree. The number of research 
questions that can be answered increased with the increase in information. This level of detail 
would allow researchers to investigate whether, for example, there are differences in 
achievement between students whose parents have associates degrees in comparison to students 
whose parents have high school diplomas, or if it is only a four year degree that makes a 
difference. 

Census data could also be used in conjunction with data from individual students, thereby 
providing additional information that students may be unable to provide. On international 
surveys it could be possible to supplement current SES measures with information from 
international census data, though access and the information collected may vary from country to 
country. 
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Figure 1. SES Measures 



Current NAEP Measures 



Aggregate Measures 



Current TIMSS 
Measures 



Educational 

Attainment 



Occupational 

Status 



Financial 

Resources 



•How far in school did your 
father go? 

•How far in school did your 
mother go? 



•% eligible for 
National School 
Lunch Program 
(School 
Questionnaire) 

•Student eligibility for 
National School 
Lunch 

Program (Admin. 
Schedule) 



•% of Census tract that has 
earned various degrees 

•HS diploma 

•BA 

•Etc. 

% of Census tract in 
each: 

•Employment status 
•Occupation 
•Industry 
•Class of worker 



•Median family income 

•% of Census tract in income 
ranges 

•Median home value 
•Mortgage status 



•How far in school did your 
father go? 

•How far in school did your 
mother go? 



• Do you have any of these 
items in your home? 



Construct Validity 

Given that the current measures of SES administered to students are often controversial 
and that new aggregate measures may not be reliable, further research is needed about the 
reliability and validity of the measurement of student SES, in particular of measurements that ask 
the students about their parents SES. Though as indicated earlier there is some indication that 
students ages 14 and 15 can answer occupational questions (Hauser, 1994; Adams & Wu, 2002). 
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The ability of students at other ages to answer these types of questions needs to be addressed. 
Similarly, the ability of students to answer questions about their parents’ education and family 
wealth needs further assessment, as does the use of aggregate data. Though aggregate measures 
of SES and composites of SES measures have benefits compared to the current measures on 
some assessments, the construct validity of these measures as well as items that students would 
answer, needs to be assessed. Construct validity is the concept that a measure accurately 
represents the concept that it is intended to measure. The true test of construct validity would be 
to assess how well these measures “predict” or explain the variance in a criterion measure. 
Similar to the validity studies done for the occupational questions in PISA 2000, a study could be 
done using the responses of parents to a series of SES questions that measure the three key SES 
constructs: Educational attainment, occupational status, and financial resources. In an ideal world 
there would be a 1 to 1 correspondence, with the proposed measures explaining all the variance 
in the criterion measure. This is unlikely to happen with the aggregate data since they are at a 
different level of analysis than the parental responses. However, the aggregate measures can be 
considered an improvement over the current measures if they explain more of the variance then 
the current NAEP measures. Similarly, if new items, such as additions to the assets list, are 
added they will need to be proven valid so that they may warrant wider inclusion. 



Conclusions 

Though the study of the measurement of socioeconomic status has a long history within 
sociology, today’s educational climate presents unique measurement needs that haven’t been 
thoroughly addressed by researchers. Though there is consensus on what factors make up SES, 
there is little guidance in the literature about what questions should be asked of students, and 
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when there is guidance, there is little proof about the ability of students to answer the questions. 
The use of Census data provides a promising alternative to asking students directly and an 
appealing alternative in the face of concerns over confidentiality and obtrusiveness. Extensive 
testing of the validity of these items and any composite measures that are created is needed. 
From this testing standards may be able to be created or “rules of thumb” that can help 
practitioners who have to balance the need for valid data with concerns about the obtrusive 
nature of SES questions. 
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